{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "56ca4d9f",
   "metadata": {
    "papermill": {
     "duration": 0.031341,
     "end_time": "2022-04-27T02:08:23.386006",
     "exception": false,
     "start_time": "2022-04-27T02:08:23.354665",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 机器读心术之神经网络与深度学习第8课书面作业\n",
    "学号：207567\n",
    "\n",
    "**书面作业：**  \n",
    "1. 试举出一个可以使用“核方法”推广至非线性情形的基础机器学习/统计学习线性算法模型的例子（如同在堂上所举的PRML中关于岭回归的例子）   \n",
    "2. 使用RBF神经网络实现课堂上“基于BP网络的个人信贷信用评估”的例子，并跟使用BP的情形对比效果。相关数据已经在“第2课课程资源”内，可使用MATLAB或其它工具软件  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beda4dd2",
   "metadata": {
    "papermill": {
     "duration": 0.028249,
     "end_time": "2022-04-27T02:08:23.442919",
     "exception": false,
     "start_time": "2022-04-27T02:08:23.414670",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 作业1\n",
    "试举出一个可以使用“核方法”推广至非线性情形的基础机器学习/统计学习线性算法模型的例子（如同在堂上所举的PRML中关于岭回归的例子）"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ca9738a",
   "metadata": {
    "papermill": {
     "duration": 0.028254,
     "end_time": "2022-04-27T02:08:23.499687",
     "exception": false,
     "start_time": "2022-04-27T02:08:23.471433",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**答：**  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffe8e0f3",
   "metadata": {
    "papermill": {
     "duration": 0.028603,
     "end_time": "2022-04-27T02:08:23.556626",
     "exception": false,
     "start_time": "2022-04-27T02:08:23.528023",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "核方法还可以应用到LDA算法模型上，LDA算法模型就是线性判别方法。  \n",
    "传统的LDA算法最后是求解如下目标函数最大化："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d51eba9b",
   "metadata": {
    "papermill": {
     "duration": 0.028168,
     "end_time": "2022-04-27T02:08:23.613204",
     "exception": false,
     "start_time": "2022-04-27T02:08:23.585036",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "$$\n",
    "J = \\frac{w^TS_bw}{w^TS_ww}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "184c84a8",
   "metadata": {
    "papermill": {
     "duration": 0.028152,
     "end_time": "2022-04-27T02:08:23.669800",
     "exception": false,
     "start_time": "2022-04-27T02:08:23.641648",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "上式中$s_b,s_w$表示类内和类间散度，它们和样本$X_i$相关。$w$是要求解的项。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a380021",
   "metadata": {
    "papermill": {
     "duration": 0.028189,
     "end_time": "2022-04-27T02:08:23.729156",
     "exception": false,
     "start_time": "2022-04-27T02:08:23.700967",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "再用上核方法后，可以将样本映射到高维空间，这样样本$\\mathbf{x}$就变成了$\\mathbf{\\phi(x)}$，欲最大化求解的目标函数变为："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "634c80f8",
   "metadata": {
    "papermill": {
     "duration": 0.02924,
     "end_time": "2022-04-27T02:08:23.789062",
     "exception": false,
     "start_time": "2022-04-27T02:08:23.759822",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "$$\n",
    "J = \\frac{w^TS^{\\phi}_bw}{w^TS^{\\phi}_ww}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a7a0d83",
   "metadata": {
    "papermill": {
     "duration": 0.028565,
     "end_time": "2022-04-27T02:08:23.847260",
     "exception": false,
     "start_time": "2022-04-27T02:08:23.818695",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "最终问题变成优化问题：\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\hat{\\mu}_0 &=\\frac{1}{m_0}\\mathbf{K1_0} \\\\\n",
    "\\hat{\\mu}_1 &=\\frac{1}{m_1}\\mathbf{K1_1} \\\\\n",
    "M&=(\\hat{\\mu}_0-\\hat{\\mu}_1)(\\hat{\\mu}_0-\\hat{\\mu}_1)^T \\\\\n",
    "N&=KK^T-\\sum_{i=0}^1 m_i\\hat{\\mu}_i\\hat{\\mu}_i^T \\\\\n",
    "\\max_{\\alpha} J(\\alpha) &= \\frac{\\alpha^TM\\alpha}{\\alpha^TN\\alpha}\n",
    "\\end{align*}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa1bee9f",
   "metadata": {
    "papermill": {
     "duration": 0.028435,
     "end_time": "2022-04-27T02:08:23.904417",
     "exception": false,
     "start_time": "2022-04-27T02:08:23.875982",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "上面系列式子中，K为核函数，其他都可以根据样本计算得到。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "039e85a2",
   "metadata": {
    "papermill": {
     "duration": 0.028145,
     "end_time": "2022-04-27T02:08:23.961052",
     "exception": false,
     "start_time": "2022-04-27T02:08:23.932907",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 作业2\n",
    "使用RBF神经网络实现课堂上“基于BP网络的个人信贷信用评估”的例子，并跟使用BP的情形对比效果。相关数据已经在“第2课课程资源”内，可使用MATLAB或其它工具软件"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "249a022b",
   "metadata": {
    "papermill": {
     "duration": 0.028428,
     "end_time": "2022-04-27T02:08:24.018592",
     "exception": false,
     "start_time": "2022-04-27T02:08:23.990164",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "说明一下作业2的环境信息：  \n",
    "1. **运行环境**：我的作业是在Kaggle上做的\n",
    "2. **数据集**：直接引用了Kaggle的德国银行信用数据集（https://www.kaggle.com/datasets/kabure/german-credit-data-with-risk） ，这个与课堂上“基于BP网络的个人信贷信用评估”的例子用的数据集出处是一致的，但是已经转化为更容易处理的形式。  \n",
    "3. **编辑语言**：我使用了python+pytorch的方式来实现本次作业2\n",
    "\n",
    "因此我同时实现了一下BP网络。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f28b0cc8",
   "metadata": {
    "papermill": {
     "duration": 0.028242,
     "end_time": "2022-04-27T02:08:24.076100",
     "exception": false,
     "start_time": "2022-04-27T02:08:24.047858",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 2.1 数据预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "eb88a24a",
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "execution": {
     "iopub.execute_input": "2022-04-27T02:08:24.137202Z",
     "iopub.status.busy": "2022-04-27T02:08:24.136520Z",
     "iopub.status.idle": "2022-04-27T02:08:24.197468Z",
     "shell.execute_reply": "2022-04-27T02:08:24.196697Z"
    },
    "papermill": {
     "duration": 0.094257,
     "end_time": "2022-04-27T02:08:24.199896",
     "exception": false,
     "start_time": "2022-04-27T02:08:24.105639",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "\n",
    "df = pd.read_csv(\"../input/german-credit-data-with-risk/german_credit_data.csv\")\n",
    "df.drop(columns=['Unnamed: 0'], inplace=True)\n",
    "obj_col = ['Sex', 'Housing', 'Saving accounts', 'Checking account', 'Purpose']\n",
    "df['Risk'] = (df['Risk']=='good').astype(int)\n",
    "df = df.fillna(0)\n",
    "df = pd.get_dummies(df, columns=obj_col)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21365acc",
   "metadata": {
    "papermill": {
     "duration": 0.029421,
     "end_time": "2022-04-27T02:08:24.258471",
     "exception": false,
     "start_time": "2022-04-27T02:08:24.229050",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "数据归一化处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f1ba3972",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-27T02:08:24.319200Z",
     "iopub.status.busy": "2022-04-27T02:08:24.318810Z",
     "iopub.status.idle": "2022-04-27T02:08:24.330015Z",
     "shell.execute_reply": "2022-04-27T02:08:24.329424Z"
    },
    "papermill": {
     "duration": 0.04417,
     "end_time": "2022-04-27T02:08:24.332074",
     "exception": false,
     "start_time": "2022-04-27T02:08:24.287904",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = (df-df.min())/(df.max()-df.min())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "342196e8",
   "metadata": {
    "papermill": {
     "duration": 0.028816,
     "end_time": "2022-04-27T02:08:24.389899",
     "exception": false,
     "start_time": "2022-04-27T02:08:24.361083",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "保存为训练数据与测试数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2f1350a6",
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "execution": {
     "iopub.execute_input": "2022-04-27T02:08:24.449196Z",
     "iopub.status.busy": "2022-04-27T02:08:24.448883Z",
     "iopub.status.idle": "2022-04-27T02:08:24.482812Z",
     "shell.execute_reply": "2022-04-27T02:08:24.482169Z"
    },
    "papermill": {
     "duration": 0.066166,
     "end_time": "2022-04-27T02:08:24.485106",
     "exception": false,
     "start_time": "2022-04-27T02:08:24.418940",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "df[:800].to_csv('train.csv',index=False)\n",
    "df[800:].to_csv('test.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "927404b5",
   "metadata": {
    "papermill": {
     "duration": 0.028356,
     "end_time": "2022-04-27T02:08:24.542374",
     "exception": false,
     "start_time": "2022-04-27T02:08:24.514018",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 2.2 建立数据集对象"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b663780f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-27T02:08:24.603487Z",
     "iopub.status.busy": "2022-04-27T02:08:24.602972Z",
     "iopub.status.idle": "2022-04-27T02:08:26.134174Z",
     "shell.execute_reply": "2022-04-27T02:08:26.133209Z"
    },
    "papermill": {
     "duration": 1.564245,
     "end_time": "2022-04-27T02:08:26.136782",
     "exception": false,
     "start_time": "2022-04-27T02:08:24.572537",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "from torchvision import transforms\n",
    "from torch.optim import Adam, SGD, RMSprop\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "74cde1b2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-27T02:08:26.197745Z",
     "iopub.status.busy": "2022-04-27T02:08:26.197066Z",
     "iopub.status.idle": "2022-04-27T02:08:26.207892Z",
     "shell.execute_reply": "2022-04-27T02:08:26.207173Z"
    },
    "papermill": {
     "duration": 0.044191,
     "end_time": "2022-04-27T02:08:26.210149",
     "exception": false,
     "start_time": "2022-04-27T02:08:26.165958",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class GermanDataSet(Dataset):\n",
    "    def __init__(self, path, transform=None):\n",
    "        self.data = pd.read_csv(path)\n",
    "        self.columns = list(self.data.columns)\n",
    "        self.label = 'Risk'\n",
    "        self.columns.remove(self.label)\n",
    "        self.transform = transform\n",
    "        \n",
    "    def __len__(self):\n",
    "        return len(self.data)\n",
    "    \n",
    "    def __getitem__(self, index):\n",
    "        item = torch.Tensor(self.data[self.columns][index:index+1].to_numpy())\n",
    "        label = torch.Tensor(self.data[self.label][index:index+1].to_numpy())\n",
    "        \n",
    "        if self.transform:\n",
    "            item = self.transform(item)\n",
    "            \n",
    "        return item, label\n",
    "    \n",
    "    def getTensor(self,samples=-1):\n",
    "        if samples==-1:\n",
    "            df = self.data[self.columns]\n",
    "        else:\n",
    "            df = self.data[self.columns].sample(samples)\n",
    "        \n",
    "        return torch.Tensor(df.to_numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6356fd74",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-27T02:08:26.270033Z",
     "iopub.status.busy": "2022-04-27T02:08:26.269411Z",
     "iopub.status.idle": "2022-04-27T02:08:26.285848Z",
     "shell.execute_reply": "2022-04-27T02:08:26.285163Z"
    },
    "papermill": {
     "duration": 0.049233,
     "end_time": "2022-04-27T02:08:26.288153",
     "exception": false,
     "start_time": "2022-04-27T02:08:26.238920",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_ds = GermanDataSet('./train.csv')\n",
    "test_ds = GermanDataSet('./test.csv')\n",
    "batch_size = 200\n",
    "train_loader = DataLoader(train_ds, batch_size=batch_size, shuffle=True)\n",
    "test_loader = DataLoader(test_ds, batch_size=batch_size, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3f6dfed",
   "metadata": {
    "papermill": {
     "duration": 0.028444,
     "end_time": "2022-04-27T02:08:26.345682",
     "exception": false,
     "start_time": "2022-04-27T02:08:26.317238",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 2.3 建立BP模型并训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5d96d806",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-27T02:08:26.404469Z",
     "iopub.status.busy": "2022-04-27T02:08:26.404090Z",
     "iopub.status.idle": "2022-04-27T02:08:26.412205Z",
     "shell.execute_reply": "2022-04-27T02:08:26.411497Z"
    },
    "papermill": {
     "duration": 0.039807,
     "end_time": "2022-04-27T02:08:26.414140",
     "exception": false,
     "start_time": "2022-04-27T02:08:26.374333",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class BPN(nn.Module):\n",
    "    def __init__(self, inputs, hiddens, outputs):\n",
    "        super(BPN, self).__init__()\n",
    "        self.model = nn.Sequential(\n",
    "            nn.Linear(inputs, hiddens),\n",
    "            nn.Sigmoid(),\n",
    "            nn.Dropout(0.5),\n",
    "            nn.Linear(hiddens, outputs),\n",
    "            nn.Sigmoid()\n",
    "        )\n",
    "        self.initialize_weights()\n",
    "        \n",
    "    def initialize_weights(self, ):\n",
    "        for _layer in self.modules():\n",
    "            if isinstance(_layer, nn.Conv2d) or isinstance(_layer, nn.Linear):\n",
    "                nn.init.xavier_normal_(_layer.weight, 2 ** 0.5)\n",
    "                \n",
    "    def forward(self, x):\n",
    "        x = self.model(x)\n",
    "        x = torch.squeeze(x, dim=1)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "bcb0e49b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-27T02:08:26.474236Z",
     "iopub.status.busy": "2022-04-27T02:08:26.473703Z",
     "iopub.status.idle": "2022-04-27T02:08:26.510759Z",
     "shell.execute_reply": "2022-04-27T02:08:26.509892Z"
    },
    "papermill": {
     "duration": 0.070119,
     "end_time": "2022-04-27T02:08:26.513380",
     "exception": false,
     "start_time": "2022-04-27T02:08:26.443261",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "bp = BPN(26, 26, 1)\n",
    "loss_fn = nn.MSELoss()\n",
    "optimizer = Adam(bp.parameters(), lr=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3dbd3c49",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-27T02:08:26.572930Z",
     "iopub.status.busy": "2022-04-27T02:08:26.572630Z",
     "iopub.status.idle": "2022-04-27T02:08:26.585417Z",
     "shell.execute_reply": "2022-04-27T02:08:26.584515Z"
    },
    "papermill": {
     "duration": 0.04526,
     "end_time": "2022-04-27T02:08:26.587679",
     "exception": false,
     "start_time": "2022-04-27T02:08:26.542419",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def train(epoch, model, fn_loss, opt, dataloader, threshold=0.5):\n",
    "    model.train()\n",
    "    accuracy , ls, count = 0, 0, 0\n",
    "    with tqdm(dataloader, unit='batch') as tepoch:\n",
    "        for X, y in tepoch:\n",
    "            tepoch.set_description(f\"epoch {epoch}\")\n",
    "            out = model(X)\n",
    "            loss = fn_loss(out, y)\n",
    "            opt.zero_grad()\n",
    "            loss.backward()\n",
    "            count += 1\n",
    "            ls += loss.item()\n",
    "            accuracy += ((out>=threshold).float() == y).sum().item()/len(X)\n",
    "            tepoch.set_postfix(loss=ls/count, accuracy='{:.3f}'.format(accuracy/count))\n",
    "\n",
    "def test(epoch, model, fn_loss, dataloader, threshold=0.5):\n",
    "    model.eval()\n",
    "    accuracy , ls, count = 0, 0, 0\n",
    "    with tqdm(dataloader, unit='batch') as tepoch:\n",
    "        for X, y in tepoch:\n",
    "            tepoch.set_description(f\"epoch {epoch}\")\n",
    "            out = model(X)\n",
    "            loss = fn_loss(out, y)\n",
    "            count += 1\n",
    "            ls += loss.item()\n",
    "            accuracy += ((out>=threshold).float() == y).sum().item()/len(X)\n",
    "            tepoch.set_postfix(loss=ls/count, accuracy='{:.3f}'.format(accuracy/count))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d7cb65e3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-27T02:08:26.647788Z",
     "iopub.status.busy": "2022-04-27T02:08:26.646977Z",
     "iopub.status.idle": "2022-04-27T02:08:29.643709Z",
     "shell.execute_reply": "2022-04-27T02:08:29.642880Z"
    },
    "papermill": {
     "duration": 3.032835,
     "end_time": "2022-04-27T02:08:29.649996",
     "exception": false,
     "start_time": "2022-04-27T02:08:26.617161",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "epoch 0: 100%|██████████| 4/4 [00:00<00:00,  5.97batch/s, accuracy=0.466, loss=0.318]\n",
      "epoch 1: 100%|██████████| 4/4 [00:00<00:00,  6.91batch/s, accuracy=0.456, loss=0.32]\n",
      "epoch 2: 100%|██████████| 4/4 [00:00<00:00,  6.99batch/s, accuracy=0.479, loss=0.314]\n",
      "epoch 3: 100%|██████████| 4/4 [00:00<00:00,  6.84batch/s, accuracy=0.451, loss=0.319]\n",
      "epoch 4: 100%|██████████| 4/4 [00:00<00:00,  7.05batch/s, accuracy=0.495, loss=0.306]\n"
     ]
    }
   ],
   "source": [
    "epochs = 5\n",
    "for i in range(epochs):\n",
    "    train(i, bp, loss_fn, optimizer, train_loader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "66230016",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-27T02:08:29.769508Z",
     "iopub.status.busy": "2022-04-27T02:08:29.768556Z",
     "iopub.status.idle": "2022-04-27T02:08:29.918705Z",
     "shell.execute_reply": "2022-04-27T02:08:29.917628Z"
    },
    "papermill": {
     "duration": 0.213224,
     "end_time": "2022-04-27T02:08:29.921404",
     "exception": false,
     "start_time": "2022-04-27T02:08:29.708180",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "epoch 0: 100%|██████████| 1/1 [00:00<00:00,  7.10batch/s, accuracy=0.410, loss=0.279]\n"
     ]
    }
   ],
   "source": [
    "test(0, bp, loss_fn, test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bbbba7f",
   "metadata": {
    "papermill": {
     "duration": 0.058826,
     "end_time": "2022-04-27T02:08:30.039080",
     "exception": false,
     "start_time": "2022-04-27T02:08:29.980254",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "测试结果大概准确率在70%左右(但是结果不稳定，有时候训练会下降到30~40%)，好像到不了课程中说的74%。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81843172",
   "metadata": {
    "papermill": {
     "duration": 0.05957,
     "end_time": "2022-04-27T02:08:30.157457",
     "exception": false,
     "start_time": "2022-04-27T02:08:30.097887",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 2.4 建立RBF模型并训练"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64ab0cb1",
   "metadata": {
    "papermill": {
     "duration": 0.057906,
     "end_time": "2022-04-27T02:08:30.275110",
     "exception": false,
     "start_time": "2022-04-27T02:08:30.217204",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "我在最后层加了一层softmax，这样将结果映射到0~1之间，方便处理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "85b199f8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-27T02:08:30.392494Z",
     "iopub.status.busy": "2022-04-27T02:08:30.392144Z",
     "iopub.status.idle": "2022-04-27T02:08:30.404967Z",
     "shell.execute_reply": "2022-04-27T02:08:30.404052Z"
    },
    "papermill": {
     "duration": 0.074001,
     "end_time": "2022-04-27T02:08:30.407104",
     "exception": false,
     "start_time": "2022-04-27T02:08:30.333103",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class RBFN(nn.Module):\n",
    "    def __init__(self, centers, n_out=3):\n",
    "        super(RBFN, self).__init__()\n",
    "        self.n_out = n_out\n",
    "        self.num_centers = centers.size(0) # 隐层节点的个数\n",
    "        self.dim_centure = centers.size(1) # \n",
    "        self.centers = nn.Parameter(centers)\n",
    "        # self.beta = nn.Parameter(torch.ones(1, self.num_centers), requires_grad=True)\n",
    "        self.beta = torch.ones(1, self.num_centers)*10\n",
    "        self.linear = nn.Linear(self.num_centers, self.n_out, bias=True)\n",
    "        self.af = nn.Softmax(dim=1)\n",
    "        self.initialize_weights()# 创建对象时自动执行\n",
    " \n",
    " \n",
    "    def kernel_fun(self, batches): #以高斯核作为径向基函数\n",
    "        n_input = batches.size(0)\n",
    "        A = self.centers.view(self.num_centers, -1).repeat(n_input, 1, 1)\n",
    "        B = batches.view(n_input, -1).unsqueeze(1).repeat(1, self.num_centers, 1)\n",
    "        C = torch.exp(-self.beta.mul((A - B).pow(2).sum(2, keepdim=False)))\n",
    "        return C\n",
    " \n",
    "    def forward(self, batches):\n",
    "        radial_val = self.kernel_fun(batches)\n",
    "        class_score = self.linear(radial_val)\n",
    "        class_score = self.af(class_score)\n",
    "        return class_score\n",
    " \n",
    "    def initialize_weights(self, ):\n",
    "        for _layer in self.modules():\n",
    "            if isinstance(_layer, nn.Conv2d) or isinstance(_layer, nn.Linear):\n",
    "                nn.init.xavier_normal_(_layer.weight, 2 ** 0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0d8f7f44",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-27T02:08:30.523687Z",
     "iopub.status.busy": "2022-04-27T02:08:30.523162Z",
     "iopub.status.idle": "2022-04-27T02:08:30.529425Z",
     "shell.execute_reply": "2022-04-27T02:08:30.528851Z"
    },
    "papermill": {
     "duration": 0.066871,
     "end_time": "2022-04-27T02:08:30.531471",
     "exception": false,
     "start_time": "2022-04-27T02:08:30.464600",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "centers = train_ds.getTensor()\n",
    "rbf = RBFN(centers,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f4960030",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-27T02:08:30.650991Z",
     "iopub.status.busy": "2022-04-27T02:08:30.650471Z",
     "iopub.status.idle": "2022-04-27T02:08:30.654707Z",
     "shell.execute_reply": "2022-04-27T02:08:30.653846Z"
    },
    "papermill": {
     "duration": 0.067681,
     "end_time": "2022-04-27T02:08:30.656725",
     "exception": false,
     "start_time": "2022-04-27T02:08:30.589044",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "loss_rbf_fn = nn.MSELoss()\n",
    "optimizer_rbf = SGD(rbf.parameters(), lr=0.001,momentum=0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a4df5e99",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-27T02:08:30.777085Z",
     "iopub.status.busy": "2022-04-27T02:08:30.776532Z",
     "iopub.status.idle": "2022-04-27T02:08:34.448711Z",
     "shell.execute_reply": "2022-04-27T02:08:34.447788Z"
    },
    "papermill": {
     "duration": 3.735466,
     "end_time": "2022-04-27T02:08:34.451210",
     "exception": false,
     "start_time": "2022-04-27T02:08:30.715744",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "epoch 0: 100%|██████████| 4/4 [00:00<00:00,  5.31batch/s, accuracy=0.701, loss=0.299]\n",
      "epoch 1: 100%|██████████| 4/4 [00:00<00:00,  5.63batch/s, accuracy=0.701, loss=0.299]\n",
      "epoch 2: 100%|██████████| 4/4 [00:00<00:00,  5.36batch/s, accuracy=0.701, loss=0.299]\n",
      "epoch 3: 100%|██████████| 4/4 [00:00<00:00,  5.61batch/s, accuracy=0.701, loss=0.299]\n",
      "epoch 4: 100%|██████████| 4/4 [00:00<00:00,  5.52batch/s, accuracy=0.701, loss=0.299]\n"
     ]
    }
   ],
   "source": [
    "epochs = 5\n",
    "for i in range(epochs):\n",
    "    train(i, rbf, loss_rbf_fn, optimizer_rbf, train_loader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ab62ef63",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-27T02:08:34.626380Z",
     "iopub.status.busy": "2022-04-27T02:08:34.625539Z",
     "iopub.status.idle": "2022-04-27T02:08:34.796898Z",
     "shell.execute_reply": "2022-04-27T02:08:34.795950Z"
    },
    "papermill": {
     "duration": 0.2629,
     "end_time": "2022-04-27T02:08:34.799254",
     "exception": false,
     "start_time": "2022-04-27T02:08:34.536354",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "epoch 0: 100%|██████████| 1/1 [00:00<00:00,  6.21batch/s, accuracy=0.695, loss=0.305]\n"
     ]
    }
   ],
   "source": [
    "test(0, rbf, loss_rbf_fn, test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "452a06d5",
   "metadata": {
    "papermill": {
     "duration": 0.09241,
     "end_time": "2022-04-27T02:08:34.980585",
     "exception": false,
     "start_time": "2022-04-27T02:08:34.888175",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "训练的结果准确率在70%左右，感觉与BP差不多，但是感觉针对这个例子，BP训练结果不稳定，或者说有时候会去到局部最小，结果只有30%左右。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 22.936952,
   "end_time": "2022-04-27T02:08:35.890165",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2022-04-27T02:08:12.953213",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
